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BACKGROUND OF T HE INVENTTON 

Field Of the Tmmnt-inn 

The invention relates generally to ultrasonic and 
radiographic medical imaging, and particularly to three- 
dimensional, ultrasonic mammography for the purpose of 
detecting and visualizing breast lesions. 

Description of the Related Ar-t- 

The standard of care for managing breast disease 
includes (in order) detection, diagnostic work-up, 
definitive diagnosis, treatment, adjuvant therapy and 
follow-up. In a large number of cases, the health care 
provider will not consider a biopsy justified and rather 
will opt to follow the situation with examinations every 
three to six months. In other cases the health care 
provider will typically also want to follow the progress of 
the lesion after adjuvant therapies are administered. This 
implies repeated imaging of the region of tissue containing 
the lesion and comparison of the earlier and later images, 
to detect and characterize any changes in the lesion. 

Various imaging modalities are available to visualize 
lesions, including X-ray (most common), ultrasonography, 
magnetic resonance imaging and other known methods. 
Whatever imaging method is used, to make meaningful 
before/after comparisons it is necessary to insure that the 
same region of tissue is under examination. Preferably the 
tissue should be projected from the same viewing angle in 
both before and after images. Comparison and even 
interpretation of images is complicated in part because 
the region of interest is a three-dimensional volume, while 
the image is a projection onto a two-dimensional plane or 
planes. Three dimensional positions and relationships must 



be inferred or calculated, based upon the available 
projections. Additional difficulties result because the 
breast may move slightly or be slightly deformed between 
the earlier and later images. Even slight misalignment 
makes it difficult to determine whether the same lesion is 
comparable in earlier and later images. Even if positions, 
compressions and projection angle correspond well in before 
and after images, side-by-side comparison of complex tissue 
patterns is difficult, and slight changes in tissue may 
escape detection. 

Summary of the invention 

An image processing system and method visually 
documents historical changes in a medical image by 
processing and comparing historical with later images. A 
composite image is created which visually emphasizes 
differences between historical and later images. 
Preferably three-dimensional, digitized images, displayable 
in various projections, are stored for archival purposes on 
computer readable media. 

To properly relate the historical and later images, an 
image processor preferably exploits an optical correlator 
to register the images accurately, by finding a 
transformation which produces a pre -determined degree of 
correlation between the images, then adjusting one image 
accordingly. The images are then compared, preferably 
volume element by volume element ("voxel by voxel") , to 
detect differences between historical and later images. A 
composite image is displayed with synthetic colors or other 
visual clues to emphasize apparent changes in the imaged 
tissue, based on differences between the historical and 
later images . 

Brief De s cri p tion of the Drawings 

FIG. 1 is a system level block diagram of an apparatus 
in accordance with the invention; 

FIG. 2a is a summary level flow diagram of a method 



for comparing historical with more recent medical imagery 
in accordance with the invention; 

FIG. 2b is a flow diagram showing the continuation of 
the method of FIG. 2a / 

FIG. 3 is a perspective view of one geometric 
arrangement which can be used to obtain ultrasonographic 
imagery of a human breast for use by the invention; 

FIG. 4 is a perspective view of the arrangement of 
FIG. 3, showing further details of a scanning scheme for 
obtaining three-dimensional image data from a human breast; 

FIG. 5a is a flow chart showing the initial steps of 
a method of registering earlier with later images, suitable 
for use in the registration step of FIG. 2a; 

FIG. 5b is a flow chart continuing from FIG. 5a, 
showing the further steps in the registration method; 

FIGs . 6a, 6b, and 6c, show simplified examples of an 
input image, a filter template, and a resulting correlation 
output image, respectively, in an example of a suitable 
cross -correlation operation which discovers positional 
offset of correlated images in the method of FIGs . 5a and 
5b; 

FIG. 7 is a perspective view of an arrangement of 
planar slices of a breast which can be selected for image 
processing of three-dimensional breast images by the 
invention; 

FIG. 8 is a perspective diagram representing a typical 
visual display of a composite image created by the 
invention to document historical changes in breast tissue; 
and 

FIG. 9 is a symbolic diagram of an optical correlator, 
optionally used in accordance with a variation of the 
invention for rapidly performing two-dimensional 
correlation operations, thereby aiding in registering the 
historical and later images. 



Detailed Description of the In vention 

Overview of Apparatus 

FIG. 1 shows an apparatus in accordance with the 
invention and suitable for practicing its method. An 
ultrasonographic imaging system 20 (or its equivalent) 
images a patient's breast 22. The imaging system 20 
provides digital image data to an image processor 24 via an 
input channel 26. An optical correlator 30 is preferably 
interfaced with and controlled by the image processor 24 to 
provide high-speed image processing (correlations) of pre- 
processed image data. A user input device 32 (typically a 
keyboard and/or graphical pointing device such as a mouse) 
is interfaced to the image processor 24 to allow user 
control of the image processor 24. Graphic output is 
displayed by the image processor 24 on a display 34, which 
is preferably a color- capable video display. A printer 3 6 
is preferably also interfaced with image processor 24 to 
produce "hard copy" printouts which record the imagery, most 
preferably with multi-color, high resolution graphics. A 
storage device 38 such as a CD-ROM writer, digital tape 
storage, DVD, or similar digital storage device should 
preferably be also interfaced with the image processor 24 
to record and store imagery in a digital data format, for 
archiving. Optionally, the image processor 24 could be 
also interfaced to a data network 39 to allow the exchange 
of data with distant users, or to access other sources of 
image data. Instead of or in addition to the data network, 
the image processor 24 can be interfaced with a digitizing 
scanner 40 which allows input of historical imagery such as 
X-ray mammograms. 

The image processor 24 is preferably a 64 bit 
workstation such as the Silicon Graphics 02, although less 
powerful processors could be used at the expense of speed 
or resolution. The ultrasonographic imaging system 20 
should preferably be capable of sweep scanning, to provide 
the multiple slices of imagery for assembly into a three- 
dimensional data set. Either a dedicated processor 



integral with the ultrasonographic imaging system 20 could 
provide such assembly, or it could alternatively be 
performed by the image processor 24 . 



5 Overview of Method 

FIGs. 2a and 2b summarize a method used in accordance 
with the invention to compare historical with later 
mammographic images. First, the image processor 24 inputs 
(step 50) historical image data. Preferably the image data 

10 is available from a data network 39, but alternatively it 

could be obtained by scanning film imagery by digitizing 
scanner 40. If available, three dimensional historical 
image data is preferred. As another option, the historical 
data could be retrieved from a mass storage device 3 8 

15 rather than loading by inputting the data. 

Next, in step 52, pathological feature data is 
preferably loaded by the image processor 24 (preferably 
from a data network but alternatively by retrieval from 
storage or through another input channel) . The 

2 0 pathological feature data is useful for narrowing the 

region of interest for image processing, and could be 
provided in any of several useful forms. Typically, a 
physical region of interest (ROI) in the historical image 
has been previously identified by a radiologist or other 

25 professional, and the coordinates of that region recorded 

in association with the historical image. In that case, 
the coordinates of the ROI are the pathological feature 
data. Alternatively, certain image shapes or 

characteristics, stored in a pathological image library, 

30 could be compared with the current image by image 

correlation to define the pathological region as described 
below in connection with FIGs. 6a -6c. A further 
alternative is to employ known automated methods for the 
detection of abnormalities in digital mammograms. For 

35 example, see U.S. Patent No. 5,4,-91,627 to Zhang et al . 

(1996) . The locations of any detected suspect lesions, as 
determined by such an automated method, could be used as 
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the pathological feature data to identify and locate the 
ROI in the image. 

Next (step 54) a coarse resolution, three-dimensional 
image sweep of the patient's breast is taken, preferably by 
5 sweeping an ultrasonic imaging head linearly across the 

breast thereby obtaining a plurality of image slices which 
collectively define a three-dimensional image. The 
resulting coarse image is stored (also in step 54) . The 
coarse breast image of step 54 is then manipulated by the 

10 image processor to properly register it (step 56) with the 

historical imagery, to at least a preliminary degree of 
registration. This initial, coarse level of registration 
can be performed by processing only a very simplified 
image, to improve processing speed. For example, the image 

15 can be first compared pixel by pixel (or voxel by voxel) to 

a threshold value, and a binary product image produced 
based on the result at each pixel (voxel) . In the 
threshold processed image, only the most obvious features, 
such as the breast outline and chest wall, will be 

20 recognizable. The images can then be rotated and shifted 

either manually (by operator command) or by the 
registration method described below in connection with 
FIGs. 5a and 5b, to crudely register the images. The 
initial registration of step 54 preferably operates on the 

25 whole image but at very low resolution: for example, the 

entire image is suitably mapped, after thresholding, onto 
a 256 by 256 matrix, then registered by the correlation 
method of FIGs. 5a and 5b. This suffices to approximately 
register the images, based on gross features only. 

3 0 Next, to obtain improved registration between the 

historical and new image, the image processor correlates 
the pathological feature set with the new image (step 58) , 
preferably by the method described below in connection with 
FIGs. 5a and 5b. The information gained from the 

35 correlation operations allows better registration of the 

new image with the historical imagery and defines an ROI in 
the new image. The ROI thus defined is then re- scanned 
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(step 60) , preferably as described above, with scanning 
ultrasound equipment but with a finer level of resolution. 
Finer resolution can be obtained by using slower scan 
speeds, higher frequency ultrasound, finer and more 
numerous receive sensors, or other techniques as known in 
the art. The fine scan, being preferably thus restricted 
to the ROI, can be executed in a shorter time than would be 
required to image the entire breast at a fine level of 
resolution. 

The image processor 24 next proceeds to register the 
new, finely resolved ROI image with the historical image of 
the corresponding ROI (step 62) . This new registration 
step 62 is performed with finer resolution than the 
previous coarse registration (step 56) , but preferably the 
operations are restricted to a smaller region of imagery, 
thus preserving a practical, limited computation time. This 
results in accurate and precise registration between the 
historical and the new tissue ROIs. Preferably, both ROIs 
are three-dimensional volume regions which are aligned by 
registration in three dimensions . 

After the historical and new images are aligned, they 
are compared in step 64 on a point -by-point basis, to 
obtain a difference image which emphasizes the temporal 
changes which have occurred. For example, if digitized two 
dimensional images are considered, the images are 
preferably first normalized to a common average value, then 
corresponding pixels are subtracted and the difference 
values stored in a corresponding position in a difference 
image. Analogously, if three-dimensional imagery is 
available (which is greatly preferred) , voxels at 
corresponding locations in the early and later images are 
subtracted one from the other, and the result stored in a 
voxel of a difference (volume) image. (A "voxel" is a unit 
of graphic information that defines a small volume element 
in three-dimensional space. It is the three-dimensional 
analog of a "pixel" which defines an area element in two- 
dimensional space.) Thus, normalized before and after 



images can be substractively compared, voxel -by - voxe 1 , to 
obtain a three-dimensional difference image which 
represents the differences between the before and after 
images . 

As a result of step 64, at least three images are 
available to the image processor 24: an early (historical) 
image; a current image; and a difference image. These 
images are then combined and/or selected for display (step 
66) in various ways, as directed by user input from input 
device 32. Finally, the resulting image or images are 
displayed (step 68) and preferably stored (step 70) and/or 
exported (step 72, preferably via the data network ) . It is 
highly preferably that the imagery be coded to emphasize 
visual differences between the earlier and later images. 
For example, the difference image can be coded in a 
distinct color for display, then superimposed on the early 
image, the later image, or a composite. Various other 
conventional methods could be equivalent ly employed to 
visually emphasize image differences. For example, in one 
variation the difference image can be displayed in 
intermittent, flickering display superimposed on either of 
the historical or current images. Significantly, the 
invention easily detects differences between the earlier 
and later images, visually displays the images, and 
visually emphasizes these differences. 

The product images from step 66 should preferably be 
recorded (step 72) for archival purposes, preferably on a 
high-density, long persistence medium such as magnetic tape 
memory or writable CD-ROM. Preferably a three-dimensional 
volume image should be stored, but to save storage space, 
operator selected projections can be derived (by 
programming the image processor 24 under control from the 
operator) and stored as two-dimensional imagery. Three 
dimensional imagery is preferred, despite its greater 
storage demands, because it can be more easily compared 
with other images (either previous history or subsequent 
development) . The three-dimensional imagery can be 



digitally manipulated by rotation, scaling, translation 
then projected to provide comparison with any view,- two 
dimensional imagery does not provide sufficient information 
to carry out the aforementioned manipulations without loss 
of information or introduction of ambiguities in the 
resulting image. The location of significant changes in the 
image should preferably be tagged with an identifier, to 
cross-link to a potential future tissue sample specimen 
number . 

The method may be repeated to compare multiple 
historical images . Either one or multiple combined images 
can be stored for archival and to document historical 
developments . 

Registration method: 

The success of the above described procedure depends 
in part upon the accuracy with which the earlier and later 
images are registered (in steps 5 6 and 62 of FIG . 2a) 
before comparison. Slight movement or deformation of the 
breast tissue or the scanning apparatus is to be expected 
between the earlier and later scans. The movement may 
include translation, rotation about any axis, or slight 
compression or expansion (in addition to biological tissue 
changes) . To adequately register the images in step, 
therefore, a computationally practical and fast method of 
registration is preferred. A preferred method of 
registration takes advantage of the specific computational 
abilities of an optical correlator (discussed in detail 
below in connection with FIG. 9) . This preferred method 
(suitable for use in step 64 of FIG. 2a ) is best described 
with reference to an example of a particular coordinate 
system, to aid in visualization. 

FIG. 3 shows how a sonographic or radiographic breast 
image might be oriented with respect to a , particular useful 
coordinate system. The system is conveniently illustrated 
in a Cartesian, rectilinear coordinate system having 
linear, perpendicular axes x, y and z, but the invention 



is not limited to such a coordinate system. The patient's 
breast 22 is scanned with ultrasound by the ultrasonic 
imaging system 2 0 (omitted for clarity, but shown in FIG. 
1) . With a patient preferably sitting facing the imaging 
system, the patient's breast 22 is preferably slightly 
compressed between pressure plates 83 (upper) and 84 
(lower) in a manner that makes maximal use of prior 
information (such as x-ray images) . In positioning the 
breast between the pressure plates, edges of the plates 
will contact the patient's chest above and below the 
breast . Because of slight movement between image 
acquisitions, the patient's breast in the imaging system, 
the axes of the later imagery do not in the general case 
exactly correspond with the x, y and z axes of the 
historical image, but may differ by a coordinate 
transformation: for example, they may differ by translation 
in the x, y or z directions, and by rotation about any 
axis. Rotation about the z axis is especially likely. 

With the breast 22 in position, ultrasonic scanning is 
preferably performed in planes (or surfaces) , which will in 
general be non-parallel to that of the Y-Z plane in FIG. 
3. FIG. 4 shows a typical geometry in which the scanhead 
9 0 includes a linear array of ultrasonic transducers 
aligned parallel to the y axis. The scanhead 90 transmits 
ultrasonic pulses in the directions of the parallel lines 
96, which are preferably perpendicular to the x'-y' plane 
and parallel to the z'-y' plane. The array of transducers 
in scanhead 90 probe the underlying tissue lying 
(approximately) on lines 96 by detecting returns of the 
ultrasonic pulses caused' by acoustic impedance 
discontinuities or reflecting surfaces within the tissue. 
The delay time between transmitting a pulse and receiving 
an return is indicative of the depth of the discontinuity 
or surface which caused the return. A characteristic such 
as magnitude, phase, or frequency of the returns is 
digitized and is plotted against the depth (z' axis) 
information and the information from the multiple 



transducers {dispersed in the y 1 direction) is assembled to 
construct an array representing a cross -sectional view of 
the tissue in a slice 98 parallel to the y'-z' plane and 
lying under scanhead 90. 
5 Multiple slices can be scanned either by providing 

multiple scanheads, a two-dimensional scanhead array, or by 
moving the scanhead across the breast, for example in the 
x' direction in FIG. 4. The planes of only a few of the 
multiple slices, specifically slices 98, 100, and 102, are 

10 shown. In practice a large number of slices is desirable, 

for better resolution. A complete set of such slices is 
preferably scanned to form a three dimensional information 
set for at least some region of interest (ROD chosen from 
the breast, which is preferably stored in a data structure 

15 (such as a three-dimensional array) to represent a three- 

dimensional image . 

Ultrasonographic equipment is available commercially 
which can be used as the ultrasonographic imaging system 2 0 
described above. A two-dimensional array of 

2 0 ultrasonographic data is usable by the invention, but with 

a diminished amount of useful information in the resulting 
display. 

Keeping in mind the exemplary coordinate system of 
FIG. 3, a method suitable for registering the historical 

25 and later images (step 64) is shown in the flow chart of 

FIG. 5a (and continued on FIG. 5b) . By this procedure the 
image processor 24 determines the proper coordinate 
transformations of scale, position, and rotation which will 
align the historical and later images. The image processor 

30 24 accesses (step 102) the stored ultrasound images from 

ultrasonographic imaging system 20 and extracts (step 104) 
a two-dimensional representation preferably by projecting 
or "collapsing" the three-dimensional ultrasound data onto 
a single plane. One method of doing this is by "cumulative 

35 projection": a projection of the three-dimensional data set 

onto a two-dimensional plane by summing the data entries 
along vectors which are perpendicular to the projection 
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plane. One such vector, vector 106, is indicated on FIG. 
4 for illustration. The density values associated with the 
voxels (three dimensional discrete volume cells) such as 
voxel 110 are summed along the vector 106. The summation of 
those density values yields a scalar value which indicates 
the sum of the tissue densities along that vector. This 
scalar value is associated with the pixel 112 at the 
intersection of the vector 106 with the x'-y' plane. 
Repeating this summation for multiple parallel vectors 
results in a set of values which defines the projection of 
the three-dimensional sonographic imagery onto the x'-y 1 
plane. This projection is preferably applied to both the 
historical and later imagery. Thus, returning to FIG. 5a, 
both three-dimensional data sets are projected onto 
respective two-dimensional images and stored (step 114) . 

The purpose of collapsing the three-dimensional images 
into two dimensions is to facilitate correlation, which is 
preferably accomplished by use of an optical correlator. 
Such correlators conventionally require two-dimensional 
imagery as correlation inputs. In some applications, the 
historical imagery will already be in two-dimensional 
projection, having been previously selected and projected 
to adequately visualize the lesion (s) of interest. In such 
a case only the later imagery must be collapsed. 

As an alternative to collapsing the images by simple 
projection, as described above, a three-dimensional image 
can be processed to create a "discriminate filter." Such 
filters are known and facilitate registration by providing 
a correlation filter which a composite of many images which 
differ by rotation about an axis. This allows testing of 
multiple, rotated slices simultaneously, by correlation 
with a composite discriminate filter. Once a discriminate 
filter is determined with acceptable correlation (at least 
equa to a pre-defined level of correlation) an individual 
filter is then selected from the set of slices which was 
used to compose the composite. In some cases, this method 
can greatly accelerate the search for best correlation. 
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The projected images are optionally further pre- 
processed (step 116) . Pre-processing 116 can include any 
of a variety of known image processing techniques including 
(without limitation) contrast modification, smoothing, 
geometric transformation, thresholding or the selection of 
a region of interest . Depending on the type of optical 
correlator 30 used, as discussed in detail below, this step 
may also include two-dimensional Fourier transformation of 
the digitized x-ray image to prepare it for subsequent 
optical correlation in a Vanderlugt optical correlator. 

Next the image processor 24 adjusts (step 118) the 
relative scale of the two images so that they better 
correspond in scale. This can be done by various methods. 
For example, one method is to match the total area of the 
cross section of the breast area between the outer outline 
and the chest wall in both images. In this method, the 
images should preferably first be processed to remove low 
contrast features, leaving only the easily visible outline 
of the breast and the chest wall . The area between these 
features in the two dimensional images is then measured, 
for example by numerical integration by the image processor 
24. The area should correspond in both images. If the 
areas do not correspond, it is possible that the degree of 
compression has changed, which can compress or expand the 
breast, v A scaling correction factor is then preferably 
applied in step 118 to correct as closely as possible. On 
the other hand, it is possible in many cases to maintain a 
relatively constant compression. In such cases, little or 
no re-scaling is necessary. 

After scale correction^ the image processor 24 
determines the rotation and translation necessary to align 
the images, preferably by interactively performing the 
steps grouped within the instruction loop 122 in FIG. 5a. 
First, two variables are initialized (step 128) : a counter 
j to control execution of an instruction loop, and an 
associated rotation angle oij . Next, a cross -correlation of 
the dual images is computed (step 130) . Preferably this 



step is accomplished by using the optical correlator 3 0 to 
perform the actual correlation computation, under control 
of the image processor 24. (The details of the optical 
correlator are discussed below, in connection with FIGs . 5 
and 6) The cross-correlation (step 130) produces a two- 
dimensional correlation output image indicating the degree 
of image cross-correlation, which is stored (step 132) 
along with the associated rotation angle a 3 . The image 
processor then checks (step 134) the counter variable to 
discover whether it has completed a prescribed number of 
iterations of the instruction loop 122 . 

Next, if the counter variable j has not reached jmax, 
the image processor 24 continues and rotates (step 136) 
one of the dual images relative to the other by some 
angular increment, for example by rotating the historical 
image one degree about an axis centered in the frame and 
parallel to the z axis. The counter is incremented (step 
138) and the procedure loops back to step 13 0 to perform 
another cross -correlation, this time with the images 
rotated incrementally. The procedure is repeated until 
some number (jmax) of differently rotated correlations has 
been performed. The parameter jmax should be chosen to be 
large enough so that the range of the associated angle 
encompasses the anticipated maximum possible range of 
rotation. For breast examination in the geometry shown in 
FIGs. 3 and 4, a range of less than 10 degrees is adequate 
in most cases. 

The rotations applied in step 13 6 are not limited to 
rotations about a single axis, but can include rotations 
about multiple independent axes (or, equivalently rotations 
about an axis obliquely oriented with respect to the 
orthogonal axes shown) . This allows the correction for an 
oblique viewing angle of one image with respect to the 
viewing angle of the other. 

After the counter j reaches jmax the image processor 
24 exits the instruction loop 122. The procedure continues 
as diagramed in FIG. 5b. The correlation output images 
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previously stored in the various iterations of step 132 are 
compared (step 140) with one another to find the 
correlation output image with the maximum correlation, and 
its associated angle of rotation a 3 . The value a m (the 
5 rotation angle which produced maximum correlation) is 

stored (step 142) and either the historical or the later 
image is rotated (step 144) by a m to bring it into the 
same orientation as its counterpart. 

It should be understood that in addition to 

10 discovering the correct scale and rotation angles, the 

cross-correlation (step 130) in the above described 
procedure produces an output image which reveals the 
correct translation (position shift) which best aligns the 
dual images. The translation information is contained in 

15 the result of the cross-correlation operation (preferably 

in two -dimensions) defined as: 

H(x, y) = g{x, y) *f(x,y) = S \f{a,/3)g{x -a,y- (})dadf3 ( 1 ) 

2 0 Where f and g are functions of two variables (images) , x 

and y are the spatial variables of the two-dimensional 
images, a, and (3 are dummy variables of integration, H is 
the cross-correlation of functions f and g, and the range 
of integration is across the entire image. If f and g 

25 differ only by a positional offset in x and y, H(x,y) will 

have a sharp peak at a position x p ,y p which is displaced 
from a central, correlation alignment position (typically 
defined as x=0, y=0) by a displacement corresponding to the 
offset between f and g. This well known result has been 

30 used to recognize and locate features of initially unknown 

locations in a field by reference to a template. See, for 
example, Russ, John C, The Image Processing Handbook (CRC 
Press, 1992), pp. 218-24. A simplified example is shown in 
FIGs. 6a- 6c. The input image, a star 146 at an offset 

35 position shown in FIG. 6a, is correlated with a filter 

derived from the centered star- shaped template 147 shown in 
FIG. 6b. The resulting correlation output shown in FIG. 6c 
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has a peak 148 at a position x p ,y p corresponding to the 
offset between the input image and the filter template . To 
align the images with a correlation peak at position x p ,y p 
it is sufficient merely to translate one of the images by 
5 a displacement equal to the offset x p ,y p . 

Returning to FIG. 5b, the image processor 24 next 
analyzes (step 154) the correlation output image to locate 
the positional offset of the correlation peak from an 
aligned correlation position, then translates (step 156) 
10 one image relative to the other as necessary to better 

align the images. After finding the optimal combination of 
rotations, scalings and translations to align the earlier 
and later images, the image processor 24 preferably stores 
(step 158) the transformed images and the transformation 
15 parameters in its associated memory and preferably outputs 

(step 160) the transformed images to a display device 34. 

In one embodiment the invention utilizes additional 
three-dimensional information about the subject body by 
further correlating the images in order to align the z 

2 0 direction (depth) . To accomplish this, the 

ultrasonographic imagery is first partitioned by image 
processor 24 into conveniently defined slices, for example 
slice 164 as shown in FIG . 7. Each slice includes one or 
more layers of the three dimensional ultrasonographic image 
25 data. The slices are defined and calculated by image 

processor 24, for example by summing data points along 
vertical vectors such as 165, to collapse multiple thin 
layers into a thicker slice (a "partial cumulative 
projection") . In FIG. 7, for example, multiple thin layers 

3 0 16 6 of ultrasonographic imagery (shown only partially to 

clarify FIG. 7) might lie between bottom slice 164 and the 
immediately overlying slice 167. The partial cumulative 
projection is taken point-by-point, by summing the 
ultrasonographic image values at points along vectors such 
35 as vector 165, and accumulating the result to the point 168 

where vector 165 intersects the slice 164. The accumulated 
data values at each defined point on slice 164 collectively 
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define the slice. 

In a typical application, slices with thickness of 5 
millimeters or less are suitable. Although planar slices 
are convenient for purposes of illustration, in some 
5 applications the slices might usefully be taken along non- 

planar contours . Such slices are also within the scope of 
the invention. Thinner slices are desirable for better 
depth definition with thin features. 

To best align the earlier and later images, 

10 corresponding slices in each are individually correlated to 

find their best registration, for example by the method 
described above. In one variation of this technique, the 
slices are defined parallel to the X'Y' plane. In another 
variation, the slices are defined in another plane (the 

15 slice under the scanhead, parallel to the Z'Y' plane, for 

example) and registered by the above described method, then 
reassembled. By registering each slice, the three- 
dimensional imagery can be aligned to account for 
variations in shear or torque between the earlier and later 

20 image acquisitions, as might be caused by variable 

deformation of the breast under varying pressure. 

The fine registration step of step 62 (in FIG. 2) is 
preferably performed by essentially the same method as the 
coarse procedure: i.e., the method of FIGs . 5a and 5b. The 

25 fine registration will yield correlation results which rely 

upon finer features than the coarse registration. Small 
constellations of microcalcif ications, in distinctive 
patterns, will yield correlation results which allow 
registration of the ROIs at a very fine scale. However, 

30 perfect correlation will not be obtained in general, 

particularly if tissue changes have occurred between the 
earlier and later images. Nevertheless, detection of 
optimum correlation should permit satisfactory 
registration. In those cases in which a dramatic tumor 

35 growth has occurred, the tissue changes will be more 

dramatically visually conspicuous, even with imperfect 
registration; contrariwise, if only slight tissue changes 
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have occurred, good correlations will be obtainable, 
resulting in precise alignment. 

After the earlier and later ROI images are adjusted to 
best register them, they can be either combined by adding, 
voxel -by- voxel, or compared by subtracting, voxel - by - voxe 1 . 
Differences between the images are easily highlighted by 
changing the color at voxels which show high variation 
between the dual images. FIG. 8 shows an example of one 
typical display of a combined image (in this case, a 
composite of earlier and later images, with differences 
highlighted) . The outline of the breast 22 is shown, with 
highlighted suspected lesion 200 (simplified for 
illustration) as revealed by a recent ultrasonogram. 
Other more static regions of density 202 and 203 are shown, 
which would suitably be displayed in a neutral color or 
grey scale. The earlier lesion region 206 is surrounded by 
the later, larger lesion 200. Suitably the later region 
200 would be highlighted, by example by display in a 
conspicuous color such as pink. The smaller lesion 206 
(visible in the earlier historical image) would preferably 
be coded in a different color (for example, green) . This 
three dimensional image, easily digitized and archived on 
computer readable media, provides easily readable 
documentation of the historical image changes, and can be 
retrieved for evidence of tissue changes. The image 
processor can also extract slices such as that cut by 
imaginary plane 3 02, and display the slice as a two- 
dimensional section for detailed inspection. Preferably, 
the processor is programmed to respond to user input so 
that any slice can be selected for display, or the user can 
sweep through multiple slices to view details of interest. 

In the procedures depicted in FIGs. 5a and 5b it is 
highly preferable that the correlation operations be 
carried out by an optical correlator. In the preferred 
embodiment, the image processor 24 electronically writes 
the dual images to the optical correlator 30. The optical 
correlator 30 preferably performs the correlation 
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operations and returns a resulting correlation image to the 
image processor 24. 

Optical correlators use wave optics to correlate 
images in two dimensions by first performing essentially a 
two-dimensional spatial Fourier transform on a two- 
dimensional source image. This method takes advantage of 
a well known mathematical property of the Fourier 
transform: many operations including correlation are more 
easily calculated in the Fourier transform domain than in 
the original spatial domain. Specifically, a two- 
dimensional correlation operation is defined by equation 1 
(above), where f(x,y) and g(x,y) are the two-dimensional 
functions or images to be cross-correlated, and a and (3 are 
dummy variables of integration. This operation can be 
performed digitally for each point x,y by numerical 
techniques, but a very large number of calculations are 
required even for one image correlation. Performing such 
an operation digitally is very time consuming and requires 
inconvenient lengths of time on any but the fastest digital 
computers . 

Unlike a conventional digital computer, an optical 
correlator can very rapidly perform a correlation 
operation, correlating a source image with a filter image 
by (1) optically Fourier transforming a source image, (2) 
comparing the source and filter image in the Fourier 
transform domain, and then (3) performing an inverse 
Fourier transformation to produce the correlation pattern 
in a spatial representation. An optical correlator can 
accomplish these operations much faster that a digital 
computer because the optical Fourier transformation is 
executed as a simultaneous operation on all points of the 
source image, using inherent properties of wave optics to 
generate the Fourier transform in two dimensions. The 
speed of the device is limited for practical purposes only 
by the available read and write speed of the data transfer 
to the correlator; the actual optical processes occur in 
fractions of a nanosecond in typical optical correlators. 
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The principles of the optical correlator are known, 
and have been described for example in the U.S. Patent No. 
5,311,359, to Lucas et al . Compact optical correlators 
suitable for use in the present invention are commercially 
5 available from Litton Data Systems, Inc., in Agoura Hills, 

California, as well as from other sources. Alternate types 
of optical correlators such as the Joint Transform 
Correlators described in U.S. Patent No. 5,650,855 to 
Kirsch et al . , U.S. Patent No. 5,216,541 to Taksue et al . 

10 or U.S. Patent No. 5,438,632 to Horner, may also be 

employed with the invention. 

For purposes of describing the present invention, the 
optical correlator 3 0 may be considered functionally as an 
electro-optical device having three (electronic) ports, as 

15 shown in FIG. 9. The three ports include: (1) an image 

input port 210 for receiving an electronic signal encoding 
an input image for correlation; (2) a filter input port 212 
for receiving a second electronic signal encoding a second 
image or "filter" for correlation; and (3) an output port 

2 0 214, typically from a charge coupled device (CCD) imager, 

which converts the correlation image into electrical form 
for output. In addition the device requires a source (not 
shown) of preferably coherent electromagnetic radiation, 
typically a laser, which provides the medium used for 
25 computation. 

Both the image input port 210 and the filter input 
port 212 are realized as two-dimensional spatial light 
modulators (SLMs) organized as two-dimensional image 
matrices, with addressable image pixels (typically arranged 

3 0 in the familiar row and column pattern) . Accordingly, the 

input image must be formatted (suitably by image processor 
24) to fit the matrix; and each pixel of data should 
preferably be addressed, under control of the image 
processor 24, to the spatially corresponding pixel on the 
35 SLM. For example, in one embodiment of the invention, the 

image input port and the filter input port are realized as 
256 x 256 pixilated matrices. Accordingly, in this 
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embodiment the image processor 24, as part of pre- 
processing step 116 (in FIG . 5a) , maps an ultrasonographic 
image onto a 256 x 256 matrix for output to the optical 
correlator 30. In a typical embodiment of the 

5 invention a Vanderlugt type optical correlator is used. In 

such a correlator the "filter" image must be pre-processed 
by two-dimensional Fourier transformation. In such an 
embodiment the image written to the filter port is 
preferably Fourier transformed by image processor 24 (for 

10 example in pre-processing step 116) , to provide a frequency 

domain pattern. In an alternate embodiment, a joint 
transform correlator may be used as optical correlator 30. 
This eliminates the need for the digital Fourier 
transformation of the filter image, as the transformation 

15 is optically performed by the joint transform correlator. 

Note that in conventional, practical high speed 
optical correlators, the SLMs and the photodetector matrix 
consist of discrete pixels rather than a continuously 
modulatable surface. Thus, the Fourier transformation and 

20 correlation operations can only approximate the discrete 

cross-correlation (given by equation 1, above) . However, 
the approximation obtained is sufficient for image 
processing for most applications. 

When the input the filter images have been written to 

25 the input and filter ports 210 and 212, the optical 

correlator produces an output image which is a two 
dimensional output correlation pattern having an optical 
peak or peaks (bright spot) at the position of greatest 
correlation between the collapsed sonographic image and the 

30 radiographic image. The degree of correlation is indicated 

by the intensity of the output signal. The position of the 
output peak on the two-dimensional matrix of the correlator 
output CCD indicates the translations or shifts of the 
images relative to one another. The output image is read 

35 from the output photodetector (CCD) 214 by the image 

processor 24 in the conventional manner, typically by 
shifting the CCD voltage values out sequentially in rows 
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(or columns) and then digitizing the output levels . 

In many applications in which two images are to be 
correlated, it is irrelevant which image provides the 
filter and which the corresponding input image, provided 
that the filter image is presented (in a Vanderlugt type 
correlator) in a Fourier transform representation. 
However, in some particular embodiments it may be 
preferable to route the historical imagery to the input 
port 210 rather than the filter port 212. For example, if 
discriminate filters are derived from the later ultrasound 
imagery and are to be correlated, these filters would 
usually be derived in a Frequency domain representation and 
would thus more appropriately routed to the Filter port 
212. On the other hand, if a search for a correlation is 
to be done with a brute-force, match search, then the 
multiple projections derived from the later imagery should 
be applied to the input port 210 (thus avoiding the need to 
perform numerous two-dimensional Fourier transformations) . 

Although the invention is described in terms of linear 
transformations of the coordinates, such as translation, 
rotation, and scale multiplication, the invention is not 
limited to linear transformations. Non-linear 
transformations of coordinate systems may be useful in some 
applications. For example, the ultrasonographic 

information may be obtained with the breast differently 
deformed, as by a change of position of the subject, or by 
instrument pressure. By applying a mathematical 

transformation, which may in general be non-linear, a 
better mapping of the deformed subject breast onto the 
original subject breast can be obtained. Similarly, some 
scanning techniques may involve curvilinear, non- cartesian 
coordinate systems which would be treated with non- linear 
transformations . 

While illustrative embodiments of the invention are 
described above, it will be evident to one skilled in the 
art that numerous variations, modifications and additional 
embodiments may be made without departing from the 



invention. For example, the construction of the 

ultrasound imaging system or the geometries and coordinate 
systems employed may be varied. Various means of data 

storage, transmission, or processing may be employed. The 
5 resolution or type of image that is sent from the image 

processor to the optical correlator could also be altered. 
Three-dimensional cross- correlations are also possible (but 
computationally complex) . To the extent that such 
operations can be decomposed into multiple planar 
10 operations, the use of the optical correlator as described 

above could greatly accelerate computations. Accordingly, 
it is intended that the invention be limited only in terms 
of the appended claims . 

15 
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WE CLAIM: 



1 . A method of visually documenting historical changes in 
biological tissue, comprising the steps of: 

(a) obtaining a first image of a region of tissue; 

(b) after obtaining said first image, obtaining a 
5 second image of said region of tissue; 

(c) digitally storing said first and second images as 
digitized first and second images ; 

(d) spatially adjusting at least one of said first and 
second digitized images to spatially register said images 

10 so that corresponding features in both images are mapped to 

corresponding positions; and 

(e) creating from said first and second digitized 
images a derived image which visually emphasizes 
differences between said first and second images, thereby 

15 visually emphasizing historical changes between said 

images . 

2. The method of claim 1, wherein said step of spatially 
adjusting at least one of said first and second digitized 
images comprises: 

determining a coordinate transformation which produces 
5 at least a pre -determined degree of correlation between 

said first and second digitized images; and 

applying said coordinate transformation to at least 
one of said first and second digitized images, to align 
said images . 

3. The method of claim 2, wherein said coordinate 
transformation is determined by: 

(a) applying a plurality of coordinate transformations 
to one of said first and second digitized images, to obtain 

5 a plurality of corresponding adjusted images; 

(b) cross-correlating said adjusted images with one of 
said first and second digitized images, to produce a 
correlation output; and 
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(c) selecting a coordinate transformation which 
10 produces at least a defined correlation output from its 

corresponding adjusted image. 

4. The method of claim 3, wherein said step of cross- 
correlating comprises: 

inputting said first and second images to an optical 
correlator, and 

5 reading said correlation output from an output of said 

optical correlator. 

5. The method of claim 2, further comprising the step of 
recording said derived image for archiving. 

6. The method of claim 5, wherein said step of recording 
comprises storing said composite image on a computer 
readable medium. 

7. The method of claim 5, wherein said step of recording 
comprises printing an image based upon said composite 
image . 

8 . The method of claim 2 , wherein at least one of said 
first and second images is a three-dimensional image. 

9. The method of claim 1, wherein said step of creating 
a composite image comprises: 

comparing an image intensity at a location in said 
first image with a respective intensity at a corresponding 
5 location in said second image, and 

determining a difference image value based upon the 
difference between said image intensity at said location in 
said first image and the respective intensity at said 
corresponding location in said second image. 
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10. The method of claim 1, wherein said composite image 
visually emphasizes image differences by representing 
various regions of said composite image in synthetic 
colors, based upon image differences between the first and 
second images . 

11. A method of creating a displayable composite 
mammographic image from a plurality of raw mammographic 
images, corresponding to earlier and later mammographic 
images, comprising the steps of: 

(a) obtaining the earlier image of a region of tissue; 

(b) obtaining the later image of substantially the 
same region of tissue ; 

(c) deriving a difference image which represents 
changes between said earlier and later images; and 

(d) combining at least one of said earlier and later 
images with said difference image, to produce a composite 
image . 

12. The method of claim 11, further comprising the step of 
recording said composite image for archiving. 

13. The method of claim 12, wherein said step of recording 
comprises storing said composite image on a computer 
readable medium. 

14. The method of claim 12, wherein said step of recording 
comprises printing an image based upon said composite 
image . 

15. The method of claim 11, wherein at least one of said 
earlier and later images is a three dimensional image 
model . 

16. The method of claim 11, wherein said step of creating 
a composite image comprises: 
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spatially adjusting at least on of said earlier and 
later images to aid in registering said images. 

17. The method of claim 16, wherein said step of spatially- 
adjusting at least one of said earlier and later images 
comprises: 

determining a coordinate transformation which produces 
5 a desired degree of correlation between said earlier and 

later images; and 

applying said coordinate transformation to at least 
one of said earlier and later images, to align said images. 

18. The method of claim 17, wherein said coordinate 
transformation is determined by: 

(a) applying a plurality of coordinate transformations 
to one of said earlier and later images, to obtain a 

5 plurality of corresponding adjusted images ,- 

(b) cross -correlating said adjusted images with one of 
said earlier and later images, to produce a correlation 
output ; and 

(c) selecting a coordinate transformation which 
10 produces a defined correlation output from its 

corresponding adjusted image. 

19 . A system for enhancing imagery of bodily tissues by 
relating earlier and later images, comprising: 

an image processor, programmed to: (a) receive said 
earlier and later images, (b) register the earlier and 
5 later images by controlling an optical correlator to find 

a position of correlation between said earlier and later 
images, and (c) derive a composite image from the earlier 
and later images ; and 

an optical correlator coupled to said image processor 
10 and arranged to correlate said earlier and later images, 

and to output to said image processor a cross correlation 
image which is indicative of the position of correlation of 
the processed images. 
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20. The system of claim 19, wherein said image processor 
is further programmed to compute differences between said 
earlier and later images and to emphasize said differences 
in said composite image. 

21. The system of claim 19, further comprising a visual 
display, coupled to said image processor and receiving from 
said image processor said composite image, to permit a user 
to view said composite image. 

22. The system of claim 19, further comprising an 
ultrasonographic imaging system, arranged to communicate 
ultrasonographic image data to said image processor to 
provide at least one of said earlier and later images. 
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HISTORICAL COMPARISON OF BREAST TISSUE BY IMAGE 
PROCESSING 



ABSTRACT OF THE INVENTION 

An image processing system and method visually 
documents and displays changes between historical and later 
mammo graphic images, preferably in three dimensions. A 
composite image is created which visually emphasizes 
5 differences between the historical and later images. 

Preferably three-dimensional, digitized images, displayable 
in various projections, are stored for archival purposes on 
computer readable media. An image processor preferably 
exploits an optical correlator to register the historical 
10 and later images accurately. The registered images are 

then compared, voxel - by - voxe 1 , to detect differences. The 
composite image is displayed with synthetic colors or other 
visual clues to emphasize apparent changes (for example, 
tumor growth or shrinkage) . 
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